perf: skip re-fetching a PR's commits when its head hasn't moved - #5
Merged
Merged
Conversation
pull_request_commits has no replication_key, so it's full_table - every time PullRequestsStream re-yields a PR (which happens on *any* change to it: a label, a comment, a review, not just new commits), this stream fully re-fetches the PR's entire commit list from page 1, forever, regardless of history already synced. For actively-discussed PRs this burns through rate limit budget on data that hasn't changed at all. A PR's commit list is immutable for a given head sha - it only changes on a new push/rebase/force-push. Track the last-synced head sha per PR (passed down via child context) and skip the fetch entirely when it's unchanged. State is partitioned per repo, so the tracking dict covers every PR in the repo without cross-PR bleed. reviews has the same full_table issue but no equivalent cheap signal (a review can be added without any commit change), so it isn't addressed here.
cedroyer
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pull_request_commitshas noreplication_key, so it'sfull_table. Every timePullRequestsStreamre-yields a PR - which happens on any change to it (a label, a comment, a review, not just new commits) - this stream fully re-fetches the PR's entire commit list from page 1, every single time, regardless of history already synced.tap-github-beekprexhausting all its GitHub tokens partway through nearly every run (round-robin summary showedprocessed 36/587 reposbefore hittingRuntimeError: All GitHub tokens have hit their rate limit).get_child_context) and skips the fetch entirely when it's unchanged. State is partitioned per repo (existingstate_partitioning_keys), so the tracking dict correctly covers every PR in the repo without cross-PR bleed.reviewshas the same full_table issue but no equivalent cheap signal (a review can be submitted without any commit change), so it isn't addressed here - flagging as a known follow-up.Test plan
ruff check/ruff format --check- clean (no new issues; pre-existing unrelated line-length issues elsewhere in the file untouched).mypy- clean.master, no regressions.